# Duelling Deep Q-Network Trading System

## Pilot Walk-Forward Scenario (6-Day Dataset)

### BASE vs VWAP State Representation Comparison

---

# 1. PROJECT DESCRIPTION

This project implements an algorithmic trading system based on Deep Reinforcement Learning.

Specifically, it employs a **Duelling Deep Q-Network (Duelling DQN)** to learn intraday trading strategies from historical stock market data.

At every time step, the agent observes the current market state and selects one of the available trading actions in order to maximize the long-term value of its portfolio.

The primary objective of this project is to investigate whether enriching the agent's state representation with **Volume Weighted Average Price (VWAP)** features improves trading performance compared with a baseline state representation, while keeping the trading environment, reward function, learning algorithm and neural network architecture unchanged.

---

# 2. DATASET

The dataset contains minute-level intraday market data for the following stocks:

- AES
- AMD
- CHK
- F

For each trading minute the dataset provides:

- Open price
- High price
- Low price
- Close price
- Trading volume
- Timestamp

The dataset consists of six consecutive trading days and is used to simulate realistic intraday trading conditions.

---

# 3. REINFORCEMENT LEARNING MODEL

The system is based on the **Duelling Deep Q-Network (Duelling DQN)** algorithm.

The objective of the agent is to estimate the action-value function

Q(s,a)

which represents the expected future reward associated with selecting action *a* in state *s*.

The neural network is divided into two parallel streams:

**Value Stream**

Estimates the value of the current state:

V(s)

**Advantage Stream**

Estimates the advantage of each available action:

A(s,a)

The final Q-value is computed as:

Q(s,a)=V(s)+(A(s,a)-mean(A(s,*)))

This implementation follows the standard Dueling DQN architecture.

---

# 4. TRADING ENVIRONMENT

The trading environment simulates an intraday financial market.

At every time step the agent receives:

- current market information,
- current portfolio state,
- current position.

The agent selects one of three possible actions:

- Hold
- Buy
- Sell

The environment continuously updates:

- portfolio value,
- open positions,
- transaction costs,
- reward,
- trading statistics.

---

# 5. PORTFOLIO INITIALIZATION

The trading environment is initialized once with an initial capital of **100,000 USD**.

The portfolio evolves continuously throughout the five Train → Trade cycles.

At the end of the episode:

- all remaining positions are closed,
- the final portfolio value is recorded.

The reported portfolio value therefore corresponds to the cumulative result obtained after completion of the fifth trading day.

---

# 6. EPISODE STRUCTURE

The experiment contains **one experimental episode** based on six consecutive trading days.

Within this episode, five consecutive **Train → Trade** cycles are executed:

Day 1 → Day 2

Day 2 → Day 3

Day 3 → Day 4

Day 4 → Day 5

Day 5 → Day 6

During each cycle:

- the agent is trained using one trading day,
- the learned policy is immediately applied to the following trading day,
- no further learning occurs during the trading day.

This walk-forward procedure prevents look-ahead bias while providing a realistic simulation of sequential trading decisions.

---

# 7. TRAIN → TRADE METHODOLOGY

The project follows a sequential Train → Trade approach.

For every cycle:

**Train Day (D)**

The agent learns from historical market data.

↓

**Trade Day (D+1)**

The learned policy is applied to the following trading day without updating the neural network parameters.

This methodology evaluates the agent on previously unseen market data while preserving chronological consistency.

---

# 8. BASE VS VWAP STATE REPRESENTATION

The system supports two alternative state representations.

## BASE configuration

The agent receives:

- normalized market variables,
- technical indicators,
- portfolio information,
- position-related variables.

## VWAP-enhanced configuration

The agent receives all BASE features together with additional VWAP-derived variables.

VWAP is computed as:

VWAP = Σ(price × volume) / Σ(volume)

The VWAP-enhanced configuration modifies **only** the state representation.

It does **not** modify:

- the trading execution mechanism,
- the reward function,
- the learning algorithm,
- the Dueling DQN architecture,
- the training procedure.

This enables a controlled comparison of the informational contribution provided by VWAP features.

---

# 9. REWARD FUNCTION

The reward function combines profitability with risk-aware performance measures.

The final reward incorporates:

- realized portfolio performance,
- Sortino ratio,
- drawdown penalty.

This formulation encourages the agent to maximize profitability while simultaneously controlling downside risk.

The reward function is identical for both BASE and VWAP configurations.

---

# 10. OUTPUT METRICS

The system records detailed execution statistics including:

- portfolio value,
- daily and cumulative rewards,
- trading activity,
- long and short transactions,
- transaction costs,
- exposure statistics,
- traded volume.

The reward function internally incorporates both the **Sortino ratio** and a **drawdown penalty**, although these quantities are not exported as standalone metrics in the output CSV files.

The recorded metrics are stored in CSV files for further analysis and visualization.

---

# 11. PROJECT OBJECTIVE

The objectives of this project are:

- develop a Dueling DQN trading agent,
- compare BASE and VWAP state representations,
- evaluate the contribution of VWAP-derived features,
- analyze the impact of state representation on trading behaviour.

---

# 12. FINAL PORTFOLIO VALUE

| Stock | BASE | VWAP |
|-------|------:|------:|
| CHK | 107,808.12 | 110,342.70 |
| AMD | 152,667.46 | 134,445.43 |
| AES | 137,907.99 | 132,105.18 |
| F | 144,347.42 | 127,764.93 |

---

# 13. MAIN FINDINGS

The experiments indicate that enriching the state representation with VWAP-derived features modifies the trading behaviour of the reinforcement learning agent but does not consistently improve overall portfolio performance.

The results suggest that adding new input features to a Deep Reinforcement Learning system does not automatically improve performance. Instead, their effectiveness depends on the additional information they contribute beyond the existing state representation.

---

# 14. NOTES

- This project corresponds to the pilot experimental scenario of the accompanying MSc thesis.
- The portfolio is initialized once at the beginning of the experiment with 100,000 USD.
- The portfolio evolves continuously throughout the five Train → Trade cycles.
- Results are fully reproducible.

---

# Author

Developed as part of an MSc thesis in Artificial Intelligence.

---

# License

This project is intended exclusively for academic and research purposes.